import warnings
warnings.filterwarnings('ignore')

8. Data exploration & Insights

8.1 Goals & Objective

• The goal here is to understand through steps, means to explore and develop intuition about unknown data
• While data may come from different sources they have several characteristics in common in terms of the data type and the quality of data
• Here we shall take a small data set and use strategies that shall generalize well for unkown data
• Usually this analysis is described a Exploratory Data Analysis (EDA) but as a business manager often you can use some of these techniques purely from a context of causal relationships and exploring cause-effect relationship

8.1.1 Variable Categories

More on Ratio versus Interval

Overall on scales and here

8.1.2 Analysis by Variable Category

8.1.3 Introducing the Analytics Scenario

Data Description:

• I found the reference to this data set here. Ames is a small city in the state of Iowa in the United States. It’s home to Iowa State University, which is the largest university in the state.

• The Ames housing dataset examines features of houses sold in Ames during the 2006–10 timeframe in and attempt to predict sale price of houses based on their features

• Usually used a academic data set to build a regression model to predict the sale prices of the houses

Domain: Real Estate

Context: zillow.com ran a competition in kaggle for a $1.2 Million prize to accurately predict price of real estate properties. In fact Zilow Reseach provides housing data for download for citizen data scientists.

The Ames data is in the same thread and very common in academic circles to explain regression and non-linearity of features.

In this class we shall use the data set to explore data and understand better how predictor variables drive/ impact target/ dependent variable; and how predictor may be related to each other.

Variable Category	Description	Column Name
Nominal	Categorical, Mutually exclusive, but not ordered, categories	MS Zoning, Street Type, Land Contour, Lot Config, Neighborhood
Ordinal	Categorical, order matters but not the difference between values	Utilities, Land Slope, Overall Qual,Overall Cond
Interval	Difference between two values is meaningful	Garage Yr Blt, Yr Sold
Ratio	All the properties of an interval variable, but also has a clear definition of 0.0	Lot Frontage, Lot Area

# standard libraries
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
# Plotting pretty figures and avoid blurry images
%config InlineBackend.figure_format = 'retina'
# Larger scale for plots in notebooks
sns.set_context('notebook')

df = pd.read_csv('https://raw.githubusercontent.com/hjhuney/Data/master/AmesHousing/train.csv', index_col=0)

For the sake of the exercise I am just going to make sure every value is populated. We shall talk in details about treatment of missing values in the next class

df.isnull().sum()[df.isnull().sum()>0]

LotFrontage      259
Alley           1369
MasVnrType         8
MasVnrArea         8
BsmtQual          37
BsmtCond          37
BsmtExposure      38
BsmtFinType1      37
BsmtFinType2      38
Electrical         1
FireplaceQu      690
GarageType        81
GarageYrBlt       81
GarageFinish      81
GarageQual        81
GarageCond        81
PoolQC          1453
Fence           1179
MiscFeature     1406
dtype: int64

df.fillna(df.median(),inplace=True)

Data Dictionary is very detailed and can be found here. Please make sure to familiarize

8.2 High Level Data Review

8.2.1 Column Description

This is a tabular data. We add the column names based on the data definition

df.shape

(1460, 80)

df.columns[df.dtypes=='object']

Index(['MSZoning', 'Street', 'Alley', 'LotShape', 'LandContour', 'Utilities',
       'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2',
       'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st',
       'Exterior2nd', 'MasVnrType', 'ExterQual', 'ExterCond', 'Foundation',
       'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2',
       'Heating', 'HeatingQC', 'CentralAir', 'Electrical', 'KitchenQual',
       'Functional', 'FireplaceQu', 'GarageType', 'GarageFinish', 'GarageQual',
       'GarageCond', 'PavedDrive', 'PoolQC', 'Fence', 'MiscFeature',
       'SaleType', 'SaleCondition'],
      dtype='object')

df.columns[df.dtypes=='int']

Index(['MSSubClass', 'LotArea', 'OverallQual', 'OverallCond', 'YearBuilt',
       'YearRemodAdd', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF',
       '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath',
       'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr',
       'TotRmsAbvGrd', 'Fireplaces', 'GarageCars', 'GarageArea', 'WoodDeckSF',
       'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea',
       'MiscVal', 'MoSold', 'YrSold', 'SalePrice'],
      dtype='object')

df.columns[df.dtypes=='float']

Index(['LotFrontage', 'MasVnrArea', 'GarageYrBlt'], dtype='object')

df.loc[:,df.columns[df.dtypes!='object']].describe().T

	count	mean	std	min	25%	50%	75%	max
MSSubClass	1460.0	56.897260	42.300571	20.0	20.00	50.0	70.00	190.0
LotFrontage	1460.0	69.863699	22.027677	21.0	60.00	69.0	79.00	313.0
LotArea	1460.0	10516.828082	9981.264932	1300.0	7553.50	9478.5	11601.50	215245.0
OverallQual	1460.0	6.099315	1.382997	1.0	5.00	6.0	7.00	10.0
OverallCond	1460.0	5.575342	1.112799	1.0	5.00	5.0	6.00	9.0
YearBuilt	1460.0	1971.267808	30.202904	1872.0	1954.00	1973.0	2000.00	2010.0
YearRemodAdd	1460.0	1984.865753	20.645407	1950.0	1967.00	1994.0	2004.00	2010.0
MasVnrArea	1460.0	103.117123	180.731373	0.0	0.00	0.0	164.25	1600.0
BsmtFinSF1	1460.0	443.639726	456.098091	0.0	0.00	383.5	712.25	5644.0
BsmtFinSF2	1460.0	46.549315	161.319273	0.0	0.00	0.0	0.00	1474.0
BsmtUnfSF	1460.0	567.240411	441.866955	0.0	223.00	477.5	808.00	2336.0
TotalBsmtSF	1460.0	1057.429452	438.705324	0.0	795.75	991.5	1298.25	6110.0
1stFlrSF	1460.0	1162.626712	386.587738	334.0	882.00	1087.0	1391.25	4692.0
2ndFlrSF	1460.0	346.992466	436.528436	0.0	0.00	0.0	728.00	2065.0
LowQualFinSF	1460.0	5.844521	48.623081	0.0	0.00	0.0	0.00	572.0
GrLivArea	1460.0	1515.463699	525.480383	334.0	1129.50	1464.0	1776.75	5642.0
BsmtFullBath	1460.0	0.425342	0.518911	0.0	0.00	0.0	1.00	3.0
BsmtHalfBath	1460.0	0.057534	0.238753	0.0	0.00	0.0	0.00	2.0
FullBath	1460.0	1.565068	0.550916	0.0	1.00	2.0	2.00	3.0
HalfBath	1460.0	0.382877	0.502885	0.0	0.00	0.0	1.00	2.0
BedroomAbvGr	1460.0	2.866438	0.815778	0.0	2.00	3.0	3.00	8.0
KitchenAbvGr	1460.0	1.046575	0.220338	0.0	1.00	1.0	1.00	3.0
TotRmsAbvGrd	1460.0	6.517808	1.625393	2.0	5.00	6.0	7.00	14.0
Fireplaces	1460.0	0.613014	0.644666	0.0	0.00	1.0	1.00	3.0
GarageYrBlt	1460.0	1978.589041	23.997022	1900.0	1962.00	1980.0	2001.00	2010.0
GarageCars	1460.0	1.767123	0.747315	0.0	1.00	2.0	2.00	4.0
GarageArea	1460.0	472.980137	213.804841	0.0	334.50	480.0	576.00	1418.0
WoodDeckSF	1460.0	94.244521	125.338794	0.0	0.00	0.0	168.00	857.0
OpenPorchSF	1460.0	46.660274	66.256028	0.0	0.00	25.0	68.00	547.0
EnclosedPorch	1460.0	21.954110	61.119149	0.0	0.00	0.0	0.00	552.0
3SsnPorch	1460.0	3.409589	29.317331	0.0	0.00	0.0	0.00	508.0
ScreenPorch	1460.0	15.060959	55.757415	0.0	0.00	0.0	0.00	480.0
PoolArea	1460.0	2.758904	40.177307	0.0	0.00	0.0	0.00	738.0
MiscVal	1460.0	43.489041	496.123024	0.0	0.00	0.0	0.00	15500.0
MoSold	1460.0	6.321918	2.703626	1.0	5.00	6.0	8.00	12.0
YrSold	1460.0	2007.815753	1.328095	2006.0	2007.00	2008.0	2009.00	2010.0
SalePrice	1460.0	180921.195890	79442.502883	34900.0	129975.00	163000.0	214000.00	755000.0

df.loc[:,df.columns[df.dtypes=='object']].describe().T

	count	unique	top	freq
MSZoning	1460	5	RL	1151
Street	1460	2	Pave	1454
Alley	91	2	Grvl	50
LotShape	1460	4	Reg	925
LandContour	1460	4	Lvl	1311
Utilities	1460	2	AllPub	1459
LotConfig	1460	5	Inside	1052
LandSlope	1460	3	Gtl	1382
Neighborhood	1460	25	NAmes	225
Condition1	1460	9	Norm	1260
Condition2	1460	8	Norm	1445
BldgType	1460	5	1Fam	1220
HouseStyle	1460	8	1Story	726
RoofStyle	1460	6	Gable	1141
RoofMatl	1460	8	CompShg	1434
Exterior1st	1460	15	VinylSd	515
Exterior2nd	1460	16	VinylSd	504
MasVnrType	1452	4	None	864
ExterQual	1460	4	TA	906
ExterCond	1460	5	TA	1282
Foundation	1460	6	PConc	647
BsmtQual	1423	4	TA	649
BsmtCond	1423	4	TA	1311
BsmtExposure	1422	4	No	953
BsmtFinType1	1423	6	Unf	430
BsmtFinType2	1422	6	Unf	1256
Heating	1460	6	GasA	1428
HeatingQC	1460	5	Ex	741
CentralAir	1460	2	Y	1365
Electrical	1459	5	SBrkr	1334
KitchenQual	1460	4	TA	735
Functional	1460	7	Typ	1360
FireplaceQu	770	5	Gd	380
GarageType	1379	6	Attchd	870
GarageFinish	1379	3	Unf	605
GarageQual	1379	5	TA	1311
GarageCond	1379	5	TA	1326
PavedDrive	1460	3	Y	1340
PoolQC	7	3	Gd	3
Fence	281	4	MnPrv	157
MiscFeature	54	4	Shed	49
SaleType	1460	9	WD	1267
SaleCondition	1460	6	Normal	1198

8.3 Target Variable

• Variable Name: SalePrice
• Variable Category: Ratio [$0 is the same as GBP 0]
• Analysis Type: Histogram, Frequencies, mean/median/mode/std., Boxplots

8.3.1 Plots & Statistics

plt.figure(figsize=(10,7))
plt.hist(df.SalePrice, color='orange', bins=200,log=False, histtype='bar')
plt.xlabel('Sale Price $')
plt.ylabel('Count')
plt.axvline(df.SalePrice.mean(), color='red', linestyle='solid', linewidth=1.5)
plt.axvline(df.SalePrice.median(), color='brown', linestyle='solid', linewidth=1.5)
plt.axvline(df.SalePrice.describe().loc['25%'], color='brown', linestyle='dashed', linewidth=1)
plt.axvline(df.SalePrice.describe().loc['75%'], color='brown', linestyle='dashed', linewidth=1)
plt.axvline(df.SalePrice.describe().loc['mean']+ 3*df.SalePrice.describe().loc['std'], color='purple', linestyle='dashed', linewidth=1)
# plt.axvline(df.SalePrice.describe().loc['mean']- 3*df.SalePrice.describe().loc['std'], color='purple', linestyle='dashed', linewidth=1)
plt.show()
df.SalePrice.describe().reset_index().T

	0	1	2	3	4	5	6	7
index	count	mean	std	min	25%	50%	75%	max
SalePrice	1460	180921	79442.5	34900	129975	163000	214000	755000

plt.figure(figsize=(10,7))
# plt.hist(df.SalePrice, color='orange', bins=200,log=False, histtype='bar')
plt.xlabel('Sale Price $')
plt.ylabel('Count')
df.SalePrice[df.SalePrice < (df.SalePrice.describe().loc['mean']+ 3*df.SalePrice.describe().loc['std'])].hist(bins=200)

<matplotlib.axes._subplots.AxesSubplot at 0x7f29e4a10a10>

Let's build a Outlier Flag- Anything larger than mean + 3*std deviation is not a outlier so out_flg == True

df['out_flg']=df.SalePrice > df.SalePrice.describe().loc['mean']+ 3*df.SalePrice.describe().loc['std']

a4_dims = (10, 7)
# df = mylib.load_data()
fig, ax = plt.subplots(figsize=a4_dims)
sns.distplot(df.SalePrice, bins=200, hist=True, kde=True, color= 'magenta', ax=ax)

<matplotlib.axes._subplots.AxesSubplot at 0x7f29e4797850>

from scipy import stats
k2, p = stats.normaltest(df.loc[df.out_flg,'SalePrice'].values)
alpha = .05
print(p)
if p < alpha:  # null hypothesis: x comes from a normal distribution
    print("The null hypothesis can be rejected")
else:
    print("The null hypothesis cannot be rejected")

0.016859801739010225
The null hypothesis can be rejected

8.3.2 Takeaways

• The Sale Price distribution is well formed and continuous
• Single mode observed
• Is skewed towards the right
• 44 sales prices are absolute outliers

8.4 Nominal Dependent Variables

• Variable Name: Neighborhood, MSZoning
• Variable Category: Nominal
• Analysis Type: Bar, frequencies, Percentages More

8.4.1 BoxPlots

sorted_nb = df.groupby(['Neighborhood'])['SalePrice'].median().sort_values(ascending= False)
sorted_nb
#dimension of the plot
a4_dims = (16, 7)
# # df = mylib.load_data()
# assign the height and width
# how to assign the dimension on a plot axis
fig, ax1 = plt.subplots(figsize=a4_dims)
ax1.set_xticklabels(ax.get_xticklabels(),rotation=90)
# , order=list(sorted_nb.index),ax=ax
### Focus on this!!!
### ax=ax is assigning the figure to a axis
sns.boxplot(x=df['Neighborhood'], y=df['SalePrice'], order=list(sorted_nb.index), ax=ax1)

<matplotlib.axes._subplots.AxesSubplot at 0x7f29e4534590>

sns.boxplot()

<matplotlib.axes._subplots.AxesSubplot at 0x7f29e452edd0>

8.4.2 Boxplot Takeaways

• We can see that in the cheapest neighborhoods houses sell for a median price of around $100,000, and in the most expensive neighborhoods houses sell for around $300,000.
• We can also see that for some neighborhoods, dispersion between the prices is very low, meaning that all the prices are close to each other.
• In the most expensive neighborhood NridgHt, however, we see a large box — there is large dispersion in the distribution of prices.

8.4.3 Violin Plot

• Violin plots are similar to box plots, except that they also show the probability density of the data at different values, usually smoothed by a kernel density estimator.

• Easy to spot multimodel disributions

sns.catplot(y="MSZoning", x="SalePrice", kind="violin", data=df, height=5, aspect=2, orient='h')

<seaborn.axisgrid.FacetGrid at 0x7f29e44cfa10>

8.4.4 Violin Plot Takeaways

• MSZoning has significant impact on variability of sale price
• C hass the lowest mean; RL & FV has the highest mean
• RL shows a significant long tail leading to a hypothesis that higher SalePrices indeed has something to do with it

8.5 Ordinal Dependent Variables

• Variable Name: Utilities, Overall Qual, Overall Cond
• Variable Category: Ordinal
• Analysis Type: Bar, Box Plot, Stem & Leaf

8.5.1 Ordered Boxplot

df.OverallCond.unique()

array([5, 8, 6, 7, 4, 2, 3, 9, 1])

#util_map= {'AllPub':5,'NoSewr':4,'NoSeWa':3,'ELO':2}
#df['util_ord']= df.Utilities.map(util_map).fillna(1)
df.OverallQual= df.OverallQual.astype(int)
df.OverallCond= df.OverallCond.astype(int)
oqc_order= np.flip(np.sort(df.OverallQual.unique(), axis=0))

np.flip(np.sort(df.OverallQual.unique(), axis=0))

array([10,  9,  8,  7,  6,  5,  4,  3,  2,  1])

oqc_order

array([10,  9,  8,  7,  6,  5,  4,  3,  2,  1])

sns.catplot(y="OverallQual", x="SalePrice", kind="box", data=df, height=5, aspect=2.5, orient='h',order= oqc_order)

<seaborn.axisgrid.FacetGrid at 0x7f29e4b3ad90>

8.5.2 Ordered Boxplot Takeaway

• As expected the mean SalePrice increases as quality improves; the relationship actually looks nonlinear
• At lower quality properties have lower variability of prices
• As quality improves it's clear that other factors play important role in explaining quality of the properties

8.5.3 Ordered Barplot

• The height of each bar represents the mean value in each category.
• Black lines at the top of each bar represent 95% confidence intervals for the mean which can be thought of as the uncertainty in our data.
• Simply put, the tips of each line are the interval where you would expect the real mean SalePrice in each category

hqC_cat= ['Po', 'Fa', 'TA','Gd', 'Ex']

sns.catplot(x='HeatingQC', y='SalePrice', data=df, kind='bar',order=hqC_cat, height=5, aspect=2)

<seaborn.axisgrid.FacetGrid at 0x7f29e4134a10>

8.5.4 Ordered Barplot Takeaways

• Little variability for poor category
• Variability increases as quality increases

8.6 Interval/ Ratio Dependent Variables

• Variable Name: 1stFlrSF, MasVnrArea,LotArea, YearBuilt
• Variable Category: First 3 Ratio; Last 1 Interval
• Analysis Type: Bar, frequencies, mean/median/mode/std, scatter

cols= ['1stFlrSF','MasVnrArea','LotArea','YearBuilt']

8.6.1 Histogram

df.loc[:,cols].hist(bins=25, figsize=(12, 12), layout=(2, 2))

array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f29e4060410>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x7f29e3fae590>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x7f29e3f61a10>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x7f29e3f17f90>]],
      dtype=object)

8.6.2 Scatterplots

a4_dims = (10, 7)
fig, ax = plt.subplots(figsize=a4_dims)
sns.scatterplot(x='LotArea', y='SalePrice', data=df,ax=ax )
# sns.scatterplot(x='LotArea', y='SalePrice', data=df.loc[df.LotArea < 50000,:],ax=ax )

<matplotlib.axes._subplots.AxesSubplot at 0x7f29e3d47290>

a4_dims = (10, 7)
fig, ax = plt.subplots(figsize=a4_dims)
# sns.scatterplot(x='LotArea', y='SalePrice', data=df,ax=ax )
sns.scatterplot(x='LotArea', y='SalePrice', data=df.loc[df.LotArea < 25000,:],ax=ax )

<matplotlib.axes._subplots.AxesSubplot at 0x7f29e3cb2890>

a4_dims = (10, 7)
fig, ax = plt.subplots(figsize=a4_dims)
# sns.scatterplot(x='LotArea', y='SalePrice', data=df,ax=ax )
sns.scatterplot(x='LotArea', y='SalePrice', data=df.loc[df.LotArea < 25000,:],ax=ax, hue='BldgType' )

<matplotlib.axes._subplots.AxesSubplot at 0x7f29e3c16a10>

8.6.3 Marginal Plots

sns.jointplot(x="LotArea", y="SalePrice",kind='hist', data=df.loc[df.LotArea < 25000,:], ax=ax, height=7)

<seaborn.axisgrid.JointGrid at 0x7f29e3ba4dd0>

8.6.4 Marginal Plots with hues

sns.jointplot(data=df.loc[df.LotArea < 25000,:], x="LotArea", y="SalePrice", hue="BldgType", height=7, kind='scatter')

<seaborn.axisgrid.JointGrid at 0x7f29e39ba890>

8.7 Composite Variable Visualization

8.7.1 .corr() cluster headmap

df.loc[:, varlst].corr()

	SalePrice	LotArea	OverallQual
SalePrice	1.000000	0.263843	0.790982
LotArea	0.263843	1.000000	0.105806
OverallQual	0.790982	0.105806	1.000000

sns.clustermap(df.loc[:, varlst].corr())

<seaborn.matrix.ClusterGrid at 0x7f29e3aaf5d0>

varlst= ['SalePrice', 'LotArea', 'BldgType', 'OverallQual', 'HeatingQC', 'MSZoning']

df.loc[3, 'SalePrice']

223500

gf=df.loc[df.LotArea < 25000,varlst]

8.7.2 .pairplot() cluster heatmap

sns.pairplot(gf, corner=True, hue='BldgType', diag_kind='hist', height=4, aspect=1)

<seaborn.axisgrid.PairGrid at 0x7f29e371c8d0>

8.7.1 .subplot() cluster headmap

fig, axes = plt.subplots(3, 3,figsize=(15,15), sharey= False)
sns.barplot(ax=axes[1,2], x=gf.BldgType, y=gf.SalePrice)
axes[1,2].set_title('SalePrice vs. Bldg Type Bar')
sns.kdeplot(ax=axes[1,1], data=gf, x="LotArea", y="SalePrice", hue="MSZoning", fill=False)
axes[1,1].set_title('SalePrice vs. MSZoning Kde')

Text(0.5, 1.0, 'SalePrice vs. MSZoning Kde')

fig, axes = plt.subplots(3, 3,figsize=(15,15), sharey= False)
## 1st row
sns.barplot(ax=axes[0,0], x=gf.BldgType, y=gf.SalePrice)
axes[0,0].set_title('SalePrice vs. Bldg Type Bar')
sns.barplot(ax=axes[0,1], x=gf.HeatingQC, y=gf.SalePrice)
axes[0,1].set_title('SalePrice vs. HeatingQC')
sns.barplot(ax=axes[0,2], x=gf.MSZoning, y=gf.SalePrice)
axes[0,2].set_title('SalePrice vs. MSZoning')
## 2nd row
sns.boxplot(ax=axes[1,0], x=gf.BldgType, y=gf.SalePrice)
axes[1,0].set_title('SalePrice vs. Bldg Type Box')
sns.boxplot(ax=axes[1,1], x=gf.HeatingQC, y=gf.SalePrice)
axes[1,1].set_title('SalePrice vs. HeatingQC Box')
sns.boxplot(ax=axes[1,2], x=gf.MSZoning, y=gf.SalePrice)
axes[1,2].set_title('SalePrice vs. MSZoning Box')
## 3rd row
sns.kdeplot(ax=axes[2,0], data=gf, x="LotArea", y="SalePrice", hue="BldgType", fill=False)
axes[2,0].set_title('SalePrice vs. Bldg Type Kde')
sns.kdeplot(ax=axes[2,1], data=gf, x="LotArea", y="SalePrice", hue="HeatingQC", fill=False)
axes[2,1].set_title('SalePrice vs. HeatingQC Kde')
sns.kdeplot(ax=axes[2,2], data=gf, x="LotArea", y="SalePrice", hue="MSZoning", fill=False)
axes[2,2].set_title('SalePrice vs. MSZoning Kde')

Text(0.5, 1.0, 'SalePrice vs. MSZoning Kde')